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ABSTRACT 

This guide was developed to provide school district, 
area vocational-technical school, and postsecondary administrators in 
Pennsylvania with guidance for managing an accountability system for 
their vocational education programs. Although drafted primarily to 
assist agencies in meeting the intent of the Carl D. Perkins 
legislation, the model presented here is designed to be qenerically 
useful for program assessment. The guide begins with a description of 
the needs assessment required for Perkins Act programs, and it 
provides information on using the needs assessment. It also provides 
guidance on using the special population needs assessment. Models a^e 
included for each type of assessment. Information on assessing 
specific uses of Perkins funds also is provided. An appendix lists 
measures and standards for various outcomes, such as participation, 
completion, retention, workplace basic skills, occupational 
achievement, human relations skill, problem solving, abstract 
reasoning, creativity, decision making emp loy ability , job success, 
and salary and wage levels. (KC) 
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REFACE 



The Cart D. Perkins Vocational and Applied Technology Act of 1990 has changed significantly the nature 

of accountability for federal vocational education funds. Whereas previous federal laws focused on fiscal 

accountability - that Is. the legality of the expenditures - THIS UW IS CLEAR IN ITS INTENT THAT 

STUDENT PERFORMANCE IMPROVE AS THE RESULT OF FEDERAL DOLLARS. 

It Is equally clear that recipients of Perkins funds must engage In rigorous evaluation procedures In order 
that: 

(1) programs needing imp'ovement can be Wentified, 

(2) the expenditures of federal funds do in fact bring about measurable improvement, and 

(3) the needs of all populations are met through quality vocational education programs. 

The assessments required for the preparation of the Local Plan were not the end. Rather they were 
simply the beginning of systematic measurement of the effectiveness of the sducational experience for 
students, it is not how many times, for how long or how often students are touched by school, but what 
students can do or do better as the resUt of that contact. The performance evaluation required by Perkins 
Is In concert with outcome assessments increasingly being required by states throughout the nation. It is 
also consistent with expectations of parents, citizens and employers who have vested Interests In how well 
schools perfom). Educators must hold themselves and each other accountable for how well students are 
prepared for their place in society. 
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Purpose 

i 

The purpose of this publication is to provide school district, area vocational-technical school arxi post- 
secorKlary administrators with guidance for managing an accountability system for their vocational education 
programs. Although drafted prirrtarHy to assist agencies to meet the Intent of the Perkins legislation, the 
paradigm presented Is designed to be generically useful for program assessment. Issues of measuring and 
documenting student performance and Identifying improvements resulting from intervention strategies are 
essential, but not unique, to the use of federal funds. 



NTRODUCTION 



In order to qualify for allocations of Perkins funds, eligible recipients were required to prepare and receive 
state approval of a Local Plan. Key to the approval of each plan was the completion of disciplined needs 
assessments to determine how well each program was performing and how well each special population 
has been served by vocational education. A labor market assessment was also required to provkJe 
additional information about training and training needs. 

Program assessment is not an event. It is an ongoing process. The above assessments provkied a 
snapshot, at one point in time, of the performance of vocational education students in programs at 
institutions contributing to the Local Plan. Although the measures may have been difficult and the 
techniques primitive at times, data were gathered that provided a base for program assessment. 

The goal of managing the Local Plan is to demonstrate measurable improvement In student performance, 
in programs and services found deficient, especially in areas impacted upon by federal funds. Techniques 
employed in the evaluation allow local program nr>anagers to separate programs and improvement strategies 
that are working from those that are not, before the deficiencies become so obvious as to impact upon the 
credibility of the system. 

The needs assessments required for the Local Plan will be the starting point of this monograph and each 
assessment will be used as stepping stones to an accountability model for local program and Local Plan 
management. 
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SING THE 
OCCUPATIONAL 
NEEDS ASSESSMENT 
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An occupational needs assessment was required in each Locai Pian. The purposes of this assessment 
were to: 

(1 ) identify where the iocai jobs are now and where they are projected to be in the foreseeable future. 

(2) identify what employers are saying about the performance of recent vocational graduates in the 
workplace. 

In conducting the occupational needs assessment, each institution was asked to limit that assessment 
to occupations relevant to their mission. Occupations conskJered not important to their mission should not 
be addressed in the assessment. Unskilled occupations, as well as occupations requiring advanced degrees 
or professional training, since they are ordinarily not included in the occupational mission, are of little direct 
interest to any applicant. Similarly, the specialization of some institutions, particularly at the postsecondary 
level, also limits the types of occupations to include in the assessment. However, by not limiting the 
assessment to occupations for which students were currently trained allowed the opportunity for discovery 
of the need for new occupational training programs. 

EvWence from this assessment provWes essential information regarding the most critical ext ernal cause 
of program failure. This evWence is part of ths overall program improvement strategy. Clearly, poor 
program performance where there is job opportunity has different meaning for program improvement than 
poor program performance where there are no jobs. 

The complexion of the iocai workforce, however, Is not static: Although change in job opportunity Is 
predictable, the frequency and sequence of change is not. Very likely, many iocai areas experience long 
periods with little change intermpted by dramatic change as major employers expand, retrench or reorganize 
their workforce. But this kind of change seldom follows an orderly timeframe. Other iocai areas may 
experience only gradual change. Schools well connected to the business community will find clues as to 
how often new occupational assessments are warranted to redocument job opportunities and skill needs. 

The second purpose of the occupational needs assessment was to determine from employers their 
perception of the abilities of vocational graduates. Although there are mixec' reviews nationally on the quality 
of occupational skills, the quality of occupational skills can be easily assessed locally on a program by 
program basis and do not appear to be the major issue. What does appear to be a national Issue is 
discontent of employers with workplace basic skills of new employees. 

Deficiencies that have been well documented from industry sun/eys include: 

(1) learning to learn; 

(2) reading, writing, and computation; 

(3) communications: listening and speaking; 

(4) creative thinking/problem solving; 
o (5) personal and career development; >j 



(6) interpersonal teamwork: 

(7) organizational effectiveness/leadership; and 

(8) others. 

This evidence does not mean, however, that every program at every school has these deficiencies. 
These are outcomes to be tested with local employers and, if deficient, Incorporated Into the improvement 
strategy. Clearly, these are extremely important aspects to the success of occupational training. Admittedly 
many of these identified deficiencies are extremely difficult to measure in an institutional setting and much 
more needs to be learned about instructional strategies to develop these skills. 



Each Local Plan also included a program and services needs assessment. The purposes of the program 
and service needs assessment were to: 

(1) Assess the success of each program and service via student participation and performance (e.g., 
access, placement, retention, achievement), 

(2) Evaluate the conditions within each program and service that contribute to the ability of each to 
achieve desired student participation and performance (e.g., curriculum, resources, equipment, 
instruction). 

Continuous systematic tracking of program performance is essential to effective program management. 
It can identify those programs in need of improvement, as well as verify the impact of implemented 
improvement strategies. Therefore, monitoring is a prerequisite to any program improvement strategy. As 
implied earlier. If there is no occupational demand for workers, no matter how well trained, resources to 
improve the program may not have positive results. However, if poor program performance cannot be 
traced to external causes, improvement of the infrastructure may be warranted. 



The purpose of assessment is to determine if programs and sen/ices are achieving the intended results 
both individually and collectively. Accountability begins by defining the outcomes a program is expected 
to achieve. In this case, outcomes are specifically stated performances expected of students because of 
their participation in school or because of specific programs or sen/ices in the school. The assessment 
model may involve process, but It begins with outcomes. Without clear outcomes it is impossible to 
determine if a program is effective. 

The key to monitoring performance of programs and services is first to select the student-centered 
outcomes that accurately and adequately reflect the expectations of each program and service. 




SING THE 

PROGRAM & SERVICE 
NEEDS ASSESSMENT 
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An outcome refers to a desirable student knowledge, skill, condition or behavtor that can be attributed 
at least In part to the educational experience and can be demonstrated it the end of that experience. Just 
as outcomes are used to plan the educational experience, they are used to assess the quality o^ that 
experience. It is essential that the outcomes used to design the program are identical to those used to 
assess its effectiveness. 

EDUCATIONAL EXPERIENCE < QIJTmMFS 

Knowledge 
Sldll 

Condition 
Behavior 

An outcome typically related to participation In vocational education Is 'employability* - a condition 
requiring certain knowledge, skills and behavior. Most programs and services have multiple outcomes. 
Some are more important than others. Examples of outcomes may be listed that reflect many of the 
purposes of vocational education programs and services. 



OUTCOI^ES 


(1) 


participation. 


(2) 


completion. 


(3) 


job retention. 


(4) 


credentialllng. 


(5) 


workplace basic skills, 


(6) 


academic skills, 


(7) 


occupational skills, 


(8) 


social skills. 


(9) 


problem solving skills. 


(10) 


abstract reasoning ability. 


(11) 


creativity. 


(12) 


employabllity, and 


(13) 


salary/wage. 



Although "participation" and "completion" are not actually outcomes, they are included in the list because 
of their relationship to program mission. This is not Intended to be a complete list. Any agreed upon set 
0; outcomes can provkJe a l^asis for what programs or services are held accountable. 

Once an outcome is specified it must be defined and the presence or absence of that outcome 
measured. The measure can verify the achievement of an outcome and also define it by describing the 
conditions under which the measurements were taken. Measurements nwy be direct, such as surveying 
students to detemtine employment status, or Indirect, such as comparing social security numbers to eamed 



income tax fUes. unompJoyment claims or welfare benefits. It may also be necessary to consider beginning 
conditions of enrollees when defining the measur«. For exaitiple, if adult enrollees are already employed 
a placement outcome may be irrelevant. A more appropriate outcome might relate to an Increase in eaming 
power. 

Outcomes and measurement are interdependent. An outcome directs the measurement but most 
outcomes way be defined and measured in nfwny different ways. 

OUTCOMES ^ MEASURE 

EmDlovabilitv ' Training relaU5d placement within 

^ ^ specified dme period 

• Gainful employment at a specified 
time following completion 

• Unemployed and looking fw employment 

Each measure may further require definltionsj, such as, "What is training related placement?", "What 
specified time pjriod?", etc. There can be no lack of clarity in specifying the measure. 

The measure and, therefore, the outcome drive the data collection efforts. How can reliable and valid 
data be collected efficiently to measure the outcome as it has been defined? Can tt.e data be aggregated 
and reported in meaningful ways? The ability to collect and use the data often influences the measures 
selected. Examples of measures of additional outcomes are appended for reference. 

Once the outcomes have been measured the results need to be interpreted, litandards are adopted to 
determine If measured achievements fall within acceptable limits. A standard is the desirable minimum level 
of student performance willing to be accepted and still term the program "successful." Any measurement 
falling below the standard is indicative of the need to improve or perhaps terminate the program or service. 
The Perkins legislation provides that programs be measured by both state and local standards. Failure to 
meet the standards requires the development of an improvement plan. Although, goals may strive for 
perfection, standards are usually something less than 100 percent achievement to allow tolerance for 
variables rtot under the control of the program. 



SUCCESSFUL 
PROGRAMS 




Standards 



UNSUCCESSFUL 
PROGRAMS 

Low 
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Standards are limited by and tied to the measures. Extension of the assessment model demonstrates 
the linear relationship from outcomes to standards. The actual standard can be any specific (usualiy 
numerical) value that represents an acceptable level of achievement under the conditions In which the 
program operated. These Illustrations are not finite n^or exhaustive of the options for selecting either 
measures or standards. 



OUTCOMES 

Employability 



MEASURES 



Training related placement 
within specified time 

Gainful employment at a 
specified time following 
completion 



Unemployed 
employment 



looking for 



STANDARDS 



Placement of a specific 
petcent ot number 

Specific percentage 
increase in the percent 
or number placed 

Specific ratio of percent 
unemployed to the local 
youth unemployment rate 



Once outcomes, measures and standards have been agreed upon, performance of indlvlduai programs 
can be periodically assessed and achievement compared to desired standards. A school lend^ itself weil 
to annual assessment because of the current semester/yearly structure. Figure 1 illustrates how 
performance measures may be applied to each program and difirerences, If any, noted between actual and 
desired performance. The magnitude of deficiencies in perfornrance or lack of progress toward achieving 
standards highlights needs for intervention in each specific program. 



Figure 1 




u 



This figure shows the data needed to assess each vocational program. Individual programs are Identified 
at the top of the chart. The measures (A, B. C. etc.) are those selecteo to assess achievement of specific 
outcomes of this program or service. The actual level of performance each year is recorded for comparison 
with established local and state standards. It may be common for programs to meet some standards and 
be deficient In others, challenging program managers to determine If Intervention strategies are warranted. 

The second purpose of the program/service needs assessment was to allow managers of local 
educational programs to evaluate internal components of each program for evidence that may contribute 
to or detract from student performance. This is a trar^sitlon from outcomes to process, since It Is presumed 
that there Is a cause-effect relationship between processes and outcomes. The typical vocational 
Instructional program may be evaluated on components, such as, curriculum, instruction, facilities and 
equipment, among others. Evaluation of each of these requires different methodologies, each beyond the 
scope of this discussion. It is the outcomes, however, that authenticate the design and redesign of the 
instructional process. 

Compared to programs, the assessment of services is considerably more complex. Services vary In 
purpose and structure. Therefore, each differs in outcomes, measures and standards. This assessment 
model may, however, be applied to service, as well as program outcomes. Each service (I.e., for special 
populations) was also assessed in the Local Plan by measures of student outcomes each service was 
specifically designed to achieve. Services not meeting expectations (with regard to achieving student 
outcomes) are analyzed to identify cause and potential strategy for modifying that service. Some of the 
more common service outcomes found in Local Plans are: 

(1) Improved academic achievement, 

(2) Improved occupational achievement, 

(3) Improved career knowledge, 

(4) Improved career decision making, 

(5) Improved retention, and 

(6) Improved placement. 

As one can see, service outcomes frequently relate to enhancement of what are commonly viewed as 
program outcomes. Often services are designed for specific populations to facilitate their ability to achieve 
exit success in a regular environment. Services outcomes may also aim directly at the program iTTgasUEfiS 
rather than the outcomes themselves in order to demonstrate improvement In measured success. A 
placement service, for example, is generally regarded as acceptable intervention that really does not Impact 
jpon the employabillty of the completers. In contrast, coaching on an exam would generally cross the line 
of acceptability. Both examples are aimed at improving measured success. Appropriate use of service 
outcomes is determined by program managers. 




SING THE 

SPECIAL POPULATION 
NEEDS ASSESSMENT 



The Perkins legislation requires needs assessments In the Local Plan for each special population. The 
special populations needs assessment posed two challenges for local administrators: 

(1) identifying the special populations. 

(2) Identifying the success each special population group is experiencing from vocational education. 
Previous federal legislation and state procedures directed energies prinfiarily toward identifying each 

special population. Definitional criteria and lists of approvable expendituies dominated the guidelines. 
Strategies and processes t)elieved to meet their needs dominated the literature. Performance was achieved 
by spending the money on significant numbers of the "right" students. 

The primary feature of current legislation is to determine whether or not each special population is 
experiencing success from participation in vocational education. The transition from criteria for identifying 
special populations to criteria to measure their success was not an easy one for program administrators to 
make. It is no longer acceptable to assume that if one identifies the special populations correctly, their 
needs are automatically known, and if exposed to well publicized solutions the result will be inherently 
successful. The success of each special population must be demonstrated at each site in terms of student 
performance. 



A 



N ASSESSMENT MODEL 

Since accountability is for measured student performance. It is appropriate to use essentially the same 
measures for any special or total population as are used in the program assessment. More emphasis may 
be required, however, among special populations to measure program access and job retention, since these 
are specific special population issues. For additional precision, one may introduce additional criteria or 
subdivide large numbers of any one special needs category (e.g., hearing impaired) in order to better tailor 
and measure the impact of specific interventions. Figure 2 illustrates the structure for this assessment. The 



SPECIAL POPtHL^ 



\( II \i ri !<) < )UM \N< I 



I'>«>| i'>«>2 l'>«>,^ 1«;'>4 




Figure 2 
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figure shows the data needed to assess the success of each special population (which Is Identified at the 
top of the figure). The measures are those selected to assess each outcome for each special population. 
The level of achievement each year on -^ach measure Is recorded for comparison with pre-established 
standards. The differences between actual performance and standards are the cues to any Improvement 
strategy. 

it nray be noted that the model for special populations Is Identical to that for programs and that most data 
needs may have already been defined. Therefore, data collected on student performance can be utilized 
to assess both the success of programs and the success of populations. 

Thus, the performance of all programs and populations can be accomplished from a single data plan. 
Data managed though a matrix configuration is illustrated in Figure 3. Each program is assessed by the 
performance of ail populations in that program. SImiiariy, the success of each population Is assessed across 
all programs. Assessment of perfomriance of specific populations within programs are facilitated where 
sufficient numbers of students allow the manager to generalize. Othenwise, the operative data analysis is 
from the total columns on each axis. 



Figure 3 




This structure of the performance assessment will insure that both programs and populations are sen/ed 
by the monitoring and accountability system. Depth of the model is achieved by the addition of measures 
and standards • vhich apply to both programs and populations. The number and complexity of the measures 
and standards beyond those established by the state are at the option of the program managers. 
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Figure 4 



Assessment like that in any other business requires a records sys ..lat can collect, access and 
analyze selected outcome measures. A history of actual student perfornr^nce provides an empirical basis 
for testing the validity and appropriateness of the uses of Perkins funds to improve programs. Only by 
documenting the actual and significant student performance can one assume that special populations have 
access to quality programs and also determine that the needs of special populations are being met through 
vocational education* 

Assessing 

/l SPECIFIC USES OF 
lA PERKINS FUNDS 

Each approved Local Plan is an agreement between a local agency and the state regarding the 
appropriate use of Perkins funds. It Is also the means by which the state has assured the federal agency 
that all state requirements of Perkins are being met and that the uses of Perkins funds are documented. 
In the Local Plan, each applicant specified the Improvements projected to occur as the osult of Perkins 
funding. These Improvements w.re based on assessed and prioritized program deficiencies. 

Therefore, In addition to applying local and state performance standards to all vocational programs and 
populations, each recipient of allocatec Perkins funds must assess progress toward meeting each objective 
specified in the approved Local Plan. The Act requires that failure to achieve satisfactory progress toward 
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those objectives by the end of the second year will require the development of an Improvement plan. The 
Improvement plan wNI consist of modifications to the planned activities and expenditures In the Local Plan 
to better achieve the proposed objectives. It may also be a n ijor consideration for the final two-year plan 
(1995-1996) applicable to the Perkins Act. 

Figure 5 Illustrates the procedures for monitoring the transHlon from the uses of funds in the Local Plan 
to the accountabHity model. The entries and Interpretations are Identical to that presented throughout this 
monograph. 

Figure 5 

Local Plan Objective — ~ — 



Specified Local 
Measures & Standards 
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The preceding discussion has focused on one paradigm for assessing the effectiveness of vocationai 
education that can be utilized fur mdnaging Local Plans. The assessments completed in order to prepare 
Local Plans were the beginning of a data based management system. 

The founc/ation of the paradigm is outcomes; these, in terms of student exiting performance, are the 
framework of program assessment. Measures are selected to determine the extent to which each outcome 
has been achieved. Measures also help define the outcome, describe the conditions under which the 
measurement is achieved, and establish the parameters for the data collection system. Standards 
are established to differentiate programs that are successful from those that are not. 

Figures and illustrations were offered to help the reader visualize the principles of the paradigm 
presented. Examples of outcomes, measures and standards are provided to assist with the explanation. 
Specific numerical examples, however, have been avoided In order to focus attention on the concepts. 

Hopefully, this discussion will contribute to the ability of program managers to improve vocationai 
education programs by better assessing the quality of programs and planning better plan intervention 
strategies where they are needed most. 
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OUTCOMES 
Paktkuwhon 



i 



iPPENDIXA 
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Examples oj measures and standards for various outcomes: 

MEASURES STANDARDS 
• Head count 



Percent ot" population! s) represented 



'f increase (decrease) 
in number enrolled 



Ratio of 
representation 
to general 
pojDulation 
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Completion 



Rktlmion 



Work Place 
Basic Skills 



()( CLPVnONAL 
ACHIKVLMLNT 



Human 

Relahons Skills 



Those completing all requirements of 
the program 

Those completing individual objectives 



Dropniiis during any specified 
nmeirame 

Those rocoivinii a passing grade 'or 
specific enrollment period(s) 

Academic skills 



% change 



• Ciu-eer knowledge 

• Portfolio assessment 

• Demonstration 

• Work Sample assessment 

• Performance tests 

• Credential or certificate exams 

• Employer ratings 

• Work samples 

• Portfolio assessment 

• Cognitive tests 

• Sociograms 

• Social adjustment inventcries 

• Employer ratings 

18 



'r chamie 



Grade levels 
standardized 
scores 

Exam scores 
Employer ratings 
Peer ratings 
Examiner ratings 



Test results 



Ratings 



Scores 



Ratings 
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/IPPENDIXA (CONTINIIKD) 



Pk()iuj:m 
Soi.viNc; 



Abstract 
Reasoning 



• Work samples 

• Employer ratings 

• Portfolio assessment 

• Tests 

• Tests 



• Ratings 

• Seores 



Ratings 




CRKArivn\' 



Decision 
Making 



Emim.oyability 



Job Success 



Salary /Wa(;I' 



• Portfolio assessment 

• Tests 

• Situation trials 



• Training related placement within 
specified time 

• Gainful employment at a specified time 
following completion 



Unemployed looking for employment 



Complete specific number 
weeks/months with same employer 
(in same job title or career ladder) 




of weeks/months after the training 
program 



Beginning salary /wage 

Increase during specified time period 



Increase over salary before training 



Ratings 



Ratings 



Placement of a 
specific % or number 

vSpecific 7r increase 
in the % or numbers 
placed 

Specific ratio of % 
unemployc J to the 
local youth 
unemployment rate 



% 



• Gainfully employed a specific number • % change 




• Amount 

♦ Comparison to 
minimum wage 

• % increase 

♦ Amount increase 
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